Asd crt/add screamv0 v1 sim data #49
Conversation
…REAM campaigns. This first commit adds a directory to document the SCREAMv0 DYAMOND2 campaign. Subsequent commits will add directories and data for the SCREAMv1 campaigns.
|
@AaronDonahue @crterai FYI I just want to iron out some details for integrating v2.1 data (#50) and then I can make any necessary adjustments in this pull request as a new commit, to keep the code/doc design consistent. (In particular, I'm planning to list the simulations in a Also, do you have any original scripts beyond |
forsyth2
force-pushed
the
ASD_CRT/add_screamv0_v1_sim_data
branch
from
62e529f to
2af396d
Compare
forsyth2
force-pushed
the
ASD_CRT/add_screamv0_v1_sim_data
branch
from
2af396d to
d38115b
Compare
forsyth2
left a comment
forsyth2
left a comment
There was a problem hiding this comment.
@AaronDonahue @crterai I think given the following:
- competing priorities at the moment
- the upcoming publication of the newsletter pointing to this data
- the
csvconversion being a bit more involved than I thought - Perlmutter being down today (which I need for automatic data size calculation)
we'll just leave this hard-coded for now. Perhaps I will clean that up when there is more time available (meaning the backend will change, but nothing will change for people viewing the data on the website).
For now though, I just made a very small commit to get the web pages to appear correctly. My latest build of the docs can be seen at https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/data_docs_49/html/.
Action items for you two:
- Please review the SCREAM pages there and tell me if everything looks as expected. If so, I'll merge this PR.
- The one original script you have isn't linked anywhere. Do you want to 1) add more original scripts? and/or 2) link the original scripts from a table?
- The FourSeasons reproduction table is largely empty; do you want to even include this table in the first place?
- Are there any ESGF links you want to include?
Table screenshots
If you're interested, I've described below the potential refactoring I had in mind, which I may return to when we're under less of a time crunch:
Proposed refactoring
The idea was to generate all the tables using a csv, because the rst files can be tricky to format correctly. The automatic table generation also has the advantage of updating everything all at once. E.g., the code calculates the data size on HPSS for each simulation.
I'm currently trying to do that with v2.1 data in #50.
However, the data provided here and structural organization of that data is a bit different from what the auto-generation code (generate_tables.py) is expecting. Notice:
| Web page | row categories (bolded rows) | simulation table columns | reproduction table columns |
|---|---|---|---|
| v2 > E3SMv2 (Water Cycle) | resolution > category |
Simulation, Data Size (TB), ESGF Links, HPSS Path | Simulation, Machine, 10 day checksum, Reproduction Script, Original Script (requires significant changes to run!!) |
| SCREAMv0 > SCREAMv0 DYAMOND2 | Simulation Name (num days) | Simulation, Data Size (TB), NERSC HPSS Path (notice missing ESGF links column) | No reproduction table |
| SCREAMv1 > SCREAMv1 Four Seasons | Simulation Name (num days) | Simulation, Data Size (GB), NERSC HPSS Path (notice missing ESGF links column) | Simulation, Machine, 10-day checksum (all empty), Reproduction Script (all empty) (notice missing Original Script column) |
Problems with extending auto-generation to this data (thus making refactoring non-trivial and hard-coding the immediate solution):
- Different organizational structure.
generate_tables.pyis expecting a rigid organizational structure (version > group > resolution > category > simulation), which doesn't appear to be matched up here, notably theresolution > categorypart. - Different columns in the tables.
- The simulations are named very differently (e.g., long name with many dots versus a month name)
|
Thanks for reviewing, @forsyth2.
The SCREAM pages look as I'd expected.
I've placed the link to the run script on the line "Scripts originally used to run SCREAMv0 simulations are available here." on this page: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/data_docs_49/html/SCREAMv0/DYAMOND2/simulation_data/index.html
Maybe we should remove it? Curious what @AaronDonahue thinks though.
We haven't published any of the SCREAM data on ESGF, so no ESGF links. |
|
Hi Ryan, thank you for taking a look at this and fixing it up a bit. In response to your questions,
Looks great to me
Do we need something like this for the FourSeasons runs as well?
Lets just remove it. I was using the v2 watercycle pages as a template but I don't think we need this for FourSeasons. If it is requested then I am happy to add it in a subsequent PR. |
|
Thanks!
@crterai Oh I missed that link. Ok, that looks good. So that one script was used to generate everything? I'm wondering if we need to include any further scripts. For v2, we listed original scripts along with reproduction scripts on https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/data_docs_49/html/v2/WaterCycle/reproducing_simulations/reproduction_table.html, but this method is fine.
@AaronDonahue That's really up to you two / the SCREAM team. I think it's probably good to have original scripts included somewhere so people can see how you generated the data, but I suppose it's not a requirement.
Ok, that sounds good. Action items:
|
|
Thanks for the follow up, @forsyth2.
Yes. We only had one 40-day production simulation with SCREAMv0, and that script was what was used to run it. |
|
@forsyth2 , I can add the run scripts I used. I am going to push a commit to the branch with them stored in the directory Can you add the link where appropriate? Like you were able to do with the SCREAMv0 links? |
|
Ok, I think everything is looking good here, so I will merge. @AaronDonahue @crterai If you do find anything you want to change, it's easy enough to open an new PR and make an adjustment, as long as you don't change the URLs you're pointing people to. |
|
@AaronDonahue @crterai The docs have now built on the actual website: https://docs.e3sm.org/e3sm_data_docs/_build/html/index.html. Please confirm everything looks good. I did check that the run script links now point to valid URLs. (This is mainly an issue on my end, but for reference, I created #51 to address the proposed refactoring I mention in #49 (review)). |
|
Thanks for pushing this through, @forsyth2. It looks good to me. |
Add provenance and data location information for the SCREAMv0 DYAMOND2 simulation and the SCREAMv1 4-Seasons simulations.